Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve dependencies management #22

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Martolivna
Copy link
Collaborator

@Martolivna Martolivna commented Feb 24, 2024

This PR enhances dependencies management with the following updates:

  • Transition to using pip --constraints for constraints handling, replacing the use of pip-compile --constraints.
  • Separate development and base requirements; base requirements are no longer included in the development requirements.
  • Update the README for the requirements installation section.
  • Update CI job's requirements installation.

These changes aim to improve clarity and reduce redundancy in the project's dependency management.

The `dev.txt` requirements file contains all needed dependencies, and
those versions already account for base requirements constraint because
it is generated by `pip-compile` with the `--constraint` option. That's
why installing dependencies from the `dev.txt` file is enough."
Updated the handling of requirements constraints
by transitioning from using `pip-compile` to directly
utilizing the `--constraints` option with `pip`.
This approach enhances clarity and reduces redundancy
by isolating dependencies.
As development requirements no longer include base requirements,
both need to be specified during installation. Using `--constraints` is
unnecessary; the requirements `.txt` files already account for it.
@@ -1,2 +1,3 @@
-r base.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally, you need to include base.in to properly layer the direct requirements. You can actually add -c base.txt inside that file, and have an additional -c dev.txt inside this one. Adding constraints from other envs might cause confusion, I think.

Copy link
Collaborator Author

@Martolivna Martolivna Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I view base requirements as core necessities, with dev (and potentially linters, prod or others later on) addressing according "additional" needs. This approach enables me to set up dependencies like building blocks, depending on use case objectives. Dependency separation is clear and understandable for me. However, I would really like to know how to do it better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This layering seems fine. Except that sometimes these things don't have to be related (like some linters may not really need all of your app deps to be installed and could be entirely separate, depending on what is set up and how).

Copy link
Member

@webknjaz webknjaz Mar 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just that you shouldn't generally entangle pip-tools' input files (with the direct dependency lists) and the output files (constraints/lockfiles). Don't make inputs of one env depend on the outputs of some other. This creates additional artificial restrictions for no good reason.

@@ -13,7 +13,7 @@ This is my educational project within IT KPI Python mentorship program by @webkn

2. Install requirements:
```console
$ pip install -r requirements/base.in -c requirements/base.txt
$ pip install -r requirements/base.txt -r requirements/dev.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The -r must use .in files, while -c should point to actual constraints. Not all of the constraints will be installable under other environments but if they aren't abused as requirements, they won't cause such problems.

Copy link
Collaborator Author

@Martolivna Martolivna Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the -r must use .in files, while -c should point to actual constraints? I've been reading articles on management to explore various solution options and also looked into dependency management at the warehouse, which also installs from .txt files. I understand that there may be factors I haven't considered, so I would really like to learn more about them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may depend on the context how they are used. I'd argue that Warehouse treats them rather simplistically.
But in general, constraints are additional limitations to how the dependency resolver should behave, while requirements are what the user said the project needs directly. Misusing them, you basically declare that your project relies on every transitive dependency and uses each directly, which is plain wrong.
There are a lot of misconceptions around this, unfortunately. And pip-tools' own docs are far from ideal, reinforcing wrong ideas.

Warehouse installs from the files with the entire tree resolved and passes --no-deps, effectively disabling the dependency resolution and telling pip to skip trying to figure out the entire tree. They also run pip check after that to verify the integrity of those things installed because otherwise you can force pip to install incompatible projects.

A more complicated problem would be visible in projects having multiple environments that are platform-dependent. When you use pip-compile under Python 3.9 on macOS, it'll likely produce some more projects in the constraint file output. It can also omit some dependencies present on other platforms. So when you attempt to install from it under Python 3.12 on GNU/Linux, you may find yourself in a situation where you demand pip to install a package which is incompatible with your environment or not installable even, exploding on pip install. It may also have to find that project that was omitted in that other env, and it wouldn't be pinned.

When you do pip install -r requirements.in, you tell pip to install projects you definitely need, and their deps. Adding -c requirements.txt may help pip find the exact concrete project dists faster and more consistently. But if it has extra entries, the won't have any effect as you don't request them to be installed unconditionally.

With multiple envs, you need multiple constraint files per each OS+arch+Python combo. But you also need to track your direct deps which you do separately. With that, you'll have pairs of two files (in+txt) for each env you identify within a project. The .in files would be layered separately and the combination of several input layers would contribute to what will end up in the unified constraint file for the target environment.

Some time ago I was trying to record the misconceptions people tend to have around lockfiles: jazzband/pip-tools#1326 (comment).

@@ -17,7 +17,7 @@ jobs:
python-version: 3.11

- name: Install requirements
run: python -m pip install -r requirements/dev.in -c requirements/dev.txt
run: python -m pip install -r requirements/base.txt -r requirements/dev.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per other comments, such implementation is unidiomatic. Use explicit -c and -r for the proper environment, without referencing other unrelated dependency groups.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants